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APOLLO SPACECRAFT RELIABILITY PROGRAM CONTROL - 
AN IMPORTANT ASPECT IN MAINTAINING 
A BALANCED RELIABILITY PROGRAM 


by J. H. Levine, Chief 
Reliability Branch 

Reliability &. Quality Assurance Division 
Manned Spacecraft Center 
Houston, Texas 


ABSTRACT 


The paper outlines reliability program control methods relating to relia- 
bility task derivation, organizational reliability responsibilities, reliabil- 
ity milestones, reliability organization manning, and the concept of closed- 
loop management control systems for task implementation. 

Methods and techniques are discussed with particular attention to the 
Apollo spacecraft reliability program. Typical problems encountered in the 
Apollo spacecraft reliability program are briefly discussed. 
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INTRODUCTION 


MSA policy dictates that "every possible practical means be employed to 
achieve high system reliability at the earliest stage of system development. " 
Frequent discussions of the disciplines required to assure reliable hardware 
have taken place over the years by technical personnel. Inevitably, the com- 
plexities of research and development (R and D) programs can lead to "after- 
the-fact" reliability programs if attention is not given to adequate control 
early in program development. For this reason, reliability program control 
methods were developed and used in the Apollo spacecraft program with specific 
attention given to: 

1. Reliability tasks based on NASA Reliability Publication NPC 250-1 en- 
titled "Reliability Program Provisions for Space System Contractors" 

2. Organizational reliability responsibilities 

3. Reliability outputs and milestones 

k. Reliability manning indexes 

5 . Closed-loop management control systems for reliability task imple- 
mentation. 

With the recognition of reliability as a vital factor in the development 
of the Apollo spacecraft, emphasis was placed on the accomplishment of relia- 
bility tasks on a timely basis. Particular attention has been given to first- 
article high reliability and to exploitation of man-in-the-loop. 

Organizational Responsibilities 

Since reliability organizations have many responsibilities, many of which 
appear redundant rather than supplementary, it was mandatory to define the rel- 
ative responsibilities of the Apollo contractors to avoid potential management 
problems. 


Reliability Funding 

The funding approach used in the Apollo spacecraft reliability program 
provides a reliability budget for only the reliability organization. The in- 
tent is to plan a total reliability program, with the reliability organization 
playing a "check-and-balance" role to assure that reliability disciplines are 
employed by all organizational elements. 

Management Control Systems 

With the development of complex manned spacecraft, the requirement for a 
well-organized management becomes mandatory. Because poor communications and 
lack of management control systems can give rise to unreliable hardware, the 
emphasis in Apollo spacecraft reliability organizations has been to develop 
and maintain a system that enhances efficient distribution and analysis of data 
and participates in program impact type tasks. 


Reliability Skills 


Although adequate program control measures are essential to a balanced 
reliability program, the importance of adequate skills and a motivated staff 
cannot be overemphasized, because these are essential to the development of re- 
liable spacecraft. 

An amplification of the above points, and methods and techniques for their 
implementation, are discussed in subsequent sections of the paper. 

ELEVEN BASIC RELIABILITY TASKS 

The need for compatible reliability program tasks was recognized. Basic 
reliability tasks were developed using the NASA Reliability Publication NFC 250-1 
as a guideline. The derivation of these tasks, together with their negotiation, 
has proven essential to efficient communicat: s with the Apollo spacecraft 

contractors, and in essence provides the framework for p lan ning the entire re- 
liability program. All of the essential reliability disciplines are included 
as a part of one or more of these tasks, which are listed below: 

1. Reliability program management 

2. Design specifications 

3- Reliability apportionment, prediction, and assessments 

4. Failure mode, effect, and criticality analysis 

5. Maintainability 

6. Design reviews 

7» Failure reporting and corrective actions 

8. Parts and materials program 

9* Test planning and monitoring 

10. Reliability indoctrination and training 

11. Reliability documentation 

A basic task-by-task boilerplate narrative is provided the contractor for 
use as a generic guide. From this boilerplate, a mutually acceptable revision 
is prepared by the Apollo spacecraft contractor within the framework of the 
above eleven tasks. These agreed-upon task descriptions are then used as the 
framework for planning and implementing the Apollo spacecraft reliability pro- 
gram. Table I summarizes the major objectives of each of the eleven reliability 
tasks. 


The Apollo spacecraft program philosophy stresses reliability organization 
in-line participation and a " check-and-balance " role in the accomplishment of 
these tasks. This role for the reliability organization has led to important 
inputs into the ultimate reliability of the Apollo spacecraft. 

As noted in table I, the tasks emphasize both qualitative and quantitative 
disciplines. The Manned Spacecraft Center has developed policies and guidelines 
that minimize dependence on reliability numerics alone. An example of this is 
the emphasis placed on crew safety, where the policy is to design subsystems on 
the basis that "no single failure shall mean loss of crew and no single failure 
shall, be cause for abort. " Application of this policy requires, during the de- 
sign process, a vigorous failure mode and effects analysis, preparation of 




reliability models, and the demonstration nonstatistically of the design ade- 
quacy via well-designed and implemented test-and-failure-reporting and 
corrective-action programs. 

TECHNIQUE FOR OBTAINING VISIBILITY IN ORGANIZATION RELIABILITY RESPONSIBILITIES 

The complexities of the Apollo spacecraft program have required the devel- 
opment of management-type systems, agreed to and enforced by top management, 
for assurance that reliability tasks are accomplished by responsible organiza- 
tions in an effective and timely manner. These systems depend upon relative 
organizational responsibilities. Typical questions requiring answers are posed, 
such as: 

1. Who is responsible for implementation of the task? 

2. Who shares task-responsibility from a participation standpoint? 

3. Who monitors the execution of each task? 

4. Who assures that the task is properly executed to obtain maximum 
reliability? 

The answer to each of these questions varies from one Apollo contractor to 
another. For example, some contractor reliability organizations are completely 
responsible for test planning, whereas other contractor reliability organiza- 
tions share this responsibility with engineering. A. matrix was developed to aid 
in assigning responsibilities (table II). It should be pointed out that the re- 
sponsibility assignments listed are typical and vary among Apollo contractors. 
The preparation of a matrix similar to that shown in table II is the initial 
step in the development of management control systems to assist in reliability 
task execution and control. 



Assurance Responsibility 

The reliability organization has "assurance responsibilities" for all of 
the tasks (table II). This requires an efficient and well-organized integrated- 
data-collection-and-analysis scheme to cope with other sections of the organi- 
zation which are responsible for separate but functionally interrelated hardware. 

Participation Responsibility 

Again, as was mentioned earlier, the emphasis in the Apollo spacecraft 
program is to utilize the reliability organization in all in-line design and 
testing functions, and at the same time to carefully avoid pre-empting specific 
responsibilities of other organizations. This delicate balance is typically 
depicted in the "participation role" shown in table II. 

To further amplify the "participation role" indicated for the reliability 
organization, reference is made to table I. Typically, the reliability organi- 
zation participates in the preparation of design specifications, hardware logic 
diagrams, failure-mode-and-effects analysis, design reviews, failure reporting 
and corrective action, and test planning and monitoring. This participation 
is considered essential in achieving the desired check-and-balance type of re- 
liability program. 




Implementation Responsibility 


The "implementation role" relative to Apollo contractor reliability organi- 
zations has typically been restricted to specialist roles, such as parts and 
materials, or to integration-type tasks, such as the failure reporting and cor- 
rective action area with respect to system design and maintenance of closeout 
actions. Another example of an integration-type role is in the assignment of 
reliability flight end- item project engineers whose responsibility is to ensure 
attention to individual flight end- items by responsible organizational elements, 
including the reliability organization. 

Monitoring Responsibility 

The "monitoring role 1, is typically restricted to those tasks where the ex- 
peditious use of resources requires a less-than- full-time role, such as in the 
selective monitoring of tests by the reliability organization. 

RELIABILITY TASK PRODUCTS AND MILESTONE DERIVATION 

Table III lists reliability tasks, typical outputs or products, and related 
milestones. These outputs must support the successful accomplishment of the 
program milestones if the reliability program is to have impact. 

Table IV depicts these milestones for a typical R and D program. The num- 
ber of reliability milestones and their interrelationships have led certain of 
the Apollo contractors to employ management control systems, such as program 
evaluation review techniques (PERT), to assist in the reliability scheduling 
activity. 

Again, as was mentioned earlier, these milestones cannot be successfully 
accomplished without continuous interface with other responsible organizational 
elements. For this reason, in the Apollo spacecraft reliability program, strong 
emphasis is placed on the data collection, analysis, and reporting scheme. The 
intent is to accomplish the objective of taking into consideration all relevant 
information to maximize successful decision-making. 

The complexities of the Apollo spacecraft development program have led to 
the requirement of key NASA/Contractor review points. A discussion of these 
review points is beyond the scope of this paper; however, they typically include 
evaluations at key design, test, and vehicle flight readiness points in the pro- 
gram. In addition to the design policies indicated earlier, first-article re- 
liability emphasis is enhanced by enforcing the NASA-Apollo spacecraft policy 
of not committing spacecraft to flight with unresolved failures or problems. 

RELIABILITY MANNING 

The agreement on manning levels for reliability program implementation can 
well be a subject for heated debate and negotiation between the customer and 
the contractor. Several important ground rules that assist in minimizing dis- 
putes were used in the Apollo spacecraft reliability program budget deter- 
mination: 









1. Reliability responsibilities delineated for organizations other than 
the reliability organization are not included in the reliability budget. 

2. The reliability organization is budgeted on the basis of their specific 
responsibilities. 

3. Reliability milestones are used to justify manning levels over the 
program span. 


Manning Ratios 

Using the above ground rules, manning estimates with a firm basis are then 
easily prepared. Apollo spacecraft program experience has shown that the ratio 
of reliability organization manning to total engineering manning ranges from 
3 to 5 percent. The specific values for a given Apollo contractor's reliability 
organization is very strongly influenced by his responsibilities. 

Typical reliability task manning values have been observed in the Apollo 
spacecraft program, as shown in table V. The major manpower allocations are to 
the reliability program management; reliability apportionment, prediction, and 
assessment; failure mode and effects analysis; failure reporting and corrective 
action; and test planning and monitoring tasks. 

Reliability Organization and Skills 

Although a detailed discussion of reliability skills and organization is 
beyond the scope of this paper, several important observations have been noted 
during the conduct of the Apollo spacecraft program: 

1. Prime consideration was given to encourage the structure of the relia- 
bility organization to preclude personnel specialization to analytical or test 
activities alone. Specialization of this type usually requires more manpower, 
added communications problems, and an organization ill-equipped to make effi- 
cient shifts with program development. 

2. Consideration was given to encourage the reliability organization to 
insure maximum flexibility in consonance with program phasing. For example, 
the organization required for the analytical phase of the program requires sig- 
nificant change when the program moves into the hardware, testing, and opera- 
tional phases. 

The acquisition of key technical skills for use in the Apollo reliability 
organizations has proven difficult due to competitive needs by other implement- 
ing parts of the contractor's organization. 

RELIABILITY MANAGEMENT CONTROL SYSTEMS 

The need for a closed- loop management control system to assure timely and 
efficient implementation of reliability tasks cannot be overemphasized. Relia- 
bility tasks, if effectively implemented, require closed-loop procedures as, 
historically, failure reporting and corrective action systems are designed to 
insure that every failure have a specific corrective action. 
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The same philosophy can he applied equally to other tasks, such as failure 
mode and effects analysis. In this instance every potential failure mode re- 
quires a closed-loop corrective-action response, such as design changes, tests, 
quality control procedures, or a combination of all three corrective actions. 
Success trends can be used, such as: (l) number of crew safety single-point 

potential failure modes, (2) number of mission success single-point potential 
failure modes, and (3) number of failure modes requiring additional quality con- 
tols. Carrying this example further, positive management procedures are then 
designed to assure that the failure mode and effects analysis is used to: 

(l) assist in proper disposition of hardware failures, (2) prepare the test pro- 
gram, (3) prepare reliability predictions, (4) prepare maintainability analyses, 
and (5) participate in design reviews. 

Ksy Management Control Elements 

The management procedures described above, although relatively simple, are 
extremely important to good reliability program control, and should take cogni- 
zance of the following four key elements: 

1. Relative organizational relationships (see table II) 

2. Data collection and analysis requirements 

3. Closeout action requirements (closed-loop requirements) 

4. Trend indicators. 

For example, these procedures can readily be applied to failure reporting 
and corrective action where the organizational responsibilities are defined 
(table II), the data collection requirement is fulfilled through procedures re- 
quiring in-line test or quality control functions to report failures, the fa: .- 
ure analysis and closeout action is fulfilled by responsible organizations, and 
indicators, such as the number of open and total failures by age and by subsys- 
tem, are used. 

Typical Management Control Subsystems 

Table VI summarizes typical management control system needs for reliability 
program control, based on the four elements discussed above. It is important to 
note that these management systems interrelate with one another, and must be 
considered during their design. 

Application of C , jed-Loop Management Procedures 

The application of closed- loop reliability management procedures varies 
with specific contractors; however, a considerable number of successful proce- 
dures have been developed and used in the Apollo spacecraft program. 

STATUS OF APOLLO SPACECRAFT RELIABILITY PROGRAM 

The previous sections of the paper briefly outlined the need for reliabil- 
ity task definition, program control, and a balanced reliability program. A 
natural question to ask is, "How has this approach worked on the Apollo space- 
craft program?" In general, the approach has been successfully used. The fol- 
lowing is a brief discussion of some of the implementation problems. 
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Derivation of Reliability Tasks and Organizational Interfaces 

In general, all of the Apollo spacecraft contractors have derived applica- 
ble versions of the reliability tasks. No real problems have arisen that de- 
tailed discussions between customer and ractor did not eliminate. The 
aspect of scheduling outputs has, to the reliability organization, had a tremen- 
dous benefit in: (l) creating an awareness of the program needs on a timely 

basis, (2) acting as a "forcing function" on other organizations relative to 
interface-type tasks, and (3) insuring efficient manpower deployment. 

The technique for displaying organizational responsibilities has provided 
benefit to organizations other than reliability in assisting in their under- 
standing of their reliability responsibilities. More effort is needed in this 
area. An effective reliability program cannot be implemented by a reliability 
organization alone. 



Reliability Maiming 

As was stated previously, the reliability manning was justified on the 
basis of specific reliability organizational responsibilities. There are ap- 
proximately 500 reliability personnel working on the Apollo spacecraft program 
at both NASA and the prime contractors. In general, problems have not been 
caused by insufficient manning, but in the shortage of specific skills, such as 
parts and materials specialists. 

Reliability Task Implementation 

Although the implementation of the Apollo spacecraft reliability program 
has had its share of the normal R and D headaches, there appears to be motiva- 
tion to get the job done in a reliable manner. In comparison to other R and D 
programs of similar complexity, the Apollo spacecraft program will leave a fine 
heritage for R and D programs that follow. 

CLOSING REMARKS 


The previous sections of the paper have pointed out: (l) the need for task 

and milestone identification, (2) the need for management control systems, and 
(3) a recognition of problems that are a normal part of complex R and Dp? ... uns 

By and large, the Apollo spacecraft reliability program has, to date, ..i 
successfully implemented and should serve in many respects as a pattern for fu- 
ture R and D programs to follow. 

A word of advice for those working in the reliability field, quoting the 
late Chester Irving Bernard, "Your responsibilities are very much greater t han 
your authority. This is the general rule for responsible people, but you may 
be misled by the current damaging half-truth ’there can be no responsibility 
without commensurate authority. *" 
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TABLE I. - RELIABILITY TASKS VERSUS MAJOR OBJECTIVES 


Task 

Major objectives 

1. Reliability program management 

1-1 Program control systems 
1-2 Audits of suocontractors 
1-3 Associate contractor support 
1-4 GFP control 

1-5 Reliability end-items project 
engineers 

1-6 Reliability plans 

2. Design specifications 

2-1 Assure specification capable of meet- 
ing reliability requirements 
2-2 Placement of numerical and other re- 
liability requirements 

3. Reliability apportionments, 
predictions, and assessments 

3-1 Reliability numerical apportionments 
3-2 Reliability numerical predictions 
3-3 Reliability numerical assessments 
3-4 Preparation of reliability models 

Failure mode, effect, and 
criticality analysis 

4-1 Failure mode and effect analysis by 
subsystem and end- item 

5. Maintainability 

5-1 Operational readiness estimates 
5-2 Design provisions for fault isolation 
arid ease of maintsnance 

6 . Design reviews 

6-1 Preparation of design review criteria 
6-2 Review of subsystem and system design 
at timely intervals using products 
indicated in other reliability tasks 

7. Failure reporting and correc- 
tive action 

7-1 Design of failure-reporting system 

7-2 Failure reportii 3 

7-3 Failure analysis 

7-4 Corrective action 

7-5 Derivation of failure trends 

8 , Farts and materials program 

8-1 Parts and materials specification 

8-2 Parts and materials selection 

8-3 Parts and materials handling and 

storage 

8-4 Parts and materials qualification 
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TABLE I.- REHABIIITY TASKS VERSUS MAJOR OBJECTIVES - Concluded I 


Task 

Major objectives 

9. Test planning and monitoring 

9-1 Test plans 

9-2 Test procedures 

9-3 Test implementation and monitoring 

9-4 Test reports 

10. Reliability indoctrination 
and training 

10-1 Program-oriented training and motiva- 
tion program 

11. Reliability documentation 

11-1 Reliability plan 
11-2 Reliability program status reports 
11-3 Failure status reports 
11-4 End-item reliability assessment 
reports 











TABLE II. - MATRIX OF RELIABILITY TASKS AND 




ORGANIZATIONAL RESPONSIBILITIES 


TYPICAL EXAMPLE 



Reliability Design 
organization engineering 


Logistics 


Quality 

control 


1. Reliability pro- 
gram management 


2. Design specifi- 
cations 


3- Reliability appor- 
tionment, predic- 
tion and assesment 


4. Failure mode and 
effects analysis 
(FMEA) 


5 . tfeinta inability 
program 


6. Design reviews 


7. Failure reporting 
and corrective 
action 


8. Parts and 

materials program 


9. Test planning and 
monitoring 


10. Reliability in- 
doctrination and 
training 


11. Reliability docu- 
mentation 


I: Implement 

P: Participate 
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TABLE III. - TYPICAL ilPOLLO SPACECRAFT RELIABILITY OUTPUTS AND MILESTONES 
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TABLE III. - TYPICAL APOLLO SPACECRAFT RELIABILITY OUTPUTS AND MILESTONES - Continued 
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TABLE III. - TYPICAL APOLLO SPACECRAFT RELIABILITY OUTPUTS AND MILESTONES - Concluded 
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TABLE IV. - TYPICAL RELIABILITY MILESTONE (A) SCHEDULE 8, 







TABLE IV. - TYPICAL RELIABILITY MILESTONE (A) SCHEDULE - Concluded 






TABLE V. - TYPICAL APOLLO SPACECRAFT MANPOWER 
ALLOCATION PER RELIABILITY TASK 


Task 

Percent of total 
reliability budget 

1. Reliability program management 

18 

2. Design specifications 

3 

3. Reliability apportionments, prediction, and 


assessments 

21 

4. Failure mode and effects analysis (FMEA) 

7 

5, Maintainability 

l 

6. Des ign , reviews 

3 

7. Failure reporting and corrective action 

25 

8. Parts and materials program 

7 

9. Test planning and monitoring 

12 

10. Reliability indoctrination and training 

1 

11. Reliability documentation 

2 


Total 100 












TABLE VI.- MAJOR RELIABILITY MANAGEMENT-CONTROL 
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MAJOR RELIABILITY MANAGEMENT-CONTROL SYSTEMS REQUIREMENTS - Concluded 
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